Predictive maintenance model design system

ABSTRACT

A data processing system for generating predictive maintenance models is disclosed, including one or more processors, a memory including one or more digital storage devices, and a plurality of instructions stored in the memory. The instructions are executable by the one or more processors to receive a historical dataset relating to each system of a plurality of systems, and including maintenance data and operational data. The instructions are further executable to receive a first selection of a first operational data feature and a first system, and display operational data associated with the first operational data feature and the first system, and maintenance data associated with the first system, on a timeline in a graphical user interface. The instructions are further executable to receive a second selection of a second operational data feature and generate a predictive maintenance model using the second operational data feature according to a machine learning method.

CROSS-REFERENCES

This application claims the benefit under 35 U.S.C. § 119(e) of thepriority of U.S. Provisional Patent Application Ser. No. 63/055,289,filed Jul. 22, 2020, the entirety of which is hereby incorporated byreference for all purposes.

BACKGROUND

Data are routinely collected by engineers on both maintenance andoperation of equipment such as the aircraft of an airline's fleet, andcan offer valuable insights into future performance and potential repairneeds. However, the complexity and sheer quantity of the collected datarenders much useful analysis beyond the skills of a typical safety ormaintenance engineer. Even for trained engineering analysts with therequisite specialist skills the process can be time consuming andlaborious using typical software tools such as Excel or Tableau.

Machine learning is an increasingly popular tool for utilizing andinterpreting such large datasets, but may be out of reach for a typicalsafety or maintenance engineer. Effective application of machinelearning techniques to a maintenance problem typically requiresidentification of a relevant data pattern or pre-cursor signature, aswell as expertise in data science to select and tune an appropriatealgorithm, and programming skills to implement training and evaluationof the algorithm to generate a predictive model.

Augmented analytics and software tools to assist in the data analysisand model design process are desirable, to bring the power and insightsof trend analysis and predictive models to a broader range of users, andsimplify and accelerate the process for experienced analysts and datascientists.

SUMMARY

The present disclosure provides systems, apparatus, and methods relatingto predictive maintenance model design. In some examples, a dataprocessing system for generating predictive maintenance models mayinclude one or more processors, a memory including one or more digitalstorage devices, and a plurality of instructions stored in the memory.The instructions may be executable by the one or more processors toreceive a historical dataset relating to each system of a plurality ofsystems, the historical dataset including maintenance data andoperational data. The instructions may be further executable to receivea first selection of a first operational data feature and a firstsystem, and display operational data associated with the firstoperational data feature and the first system, and maintenance dataassociated with the first system, on a timeline in a graphical userinterface. The instructions may be further executable to receive asecond selection of a second operational data feature and generate apredictive maintenance model using the second operational data featureaccording to a machine learning method.

In some examples, a computer implemented method of generating apredictive maintenance model may include receiving a historical datasetrelating to each system of a plurality of systems, the historicaldataset including maintenance data and operational data. The method mayfurther include receiving a first selection of a first operational datafeature and a first system, and displaying operational data associatedwith the first operational data feature and the first system, andmaintenance data associated with the first system, on a timeline in agraphical user interface. The method may further include receiving asecond selection of a second operational data feature and generating apredictive maintenance model using the second operational data featureaccording to a machine learning method.

In some examples, a computer program product for generating predictivemaintenance models may include a non-transitory computer-readablestorage medium having computer-readable program code embodied in thestorage medium, the computer-readable program code configured to cause adata processing system to generate a predictive maintenance model. Thecode may include at least one instruction to receive a historicaldataset relating to each system of a plurality of systems, thehistorical dataset including maintenance data and operational data. Thecode may further include at least one instruction to receive a firstselection of a first operational data feature and a first system, anddisplay operational data associated with the first operational datafeature and the first system and maintenance data associated with thefirst system, on a timeline in a graphical user interface. The code mayinclude at least one instruction to receive a second selection of asecond operational data feature and generate a predictive maintenancemodel using the second operational data feature according to a machinelearning method.

Features, functions, and advantages may be achieved independently invarious examples of the present disclosure, or may be combined in yetother examples, further details of which can be seen with reference tothe following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart depicting steps of an illustrative predictivemaintenance model design process in accordance with aspects of thepresent disclosure.

FIG. 2 is a schematic diagram of an illustrative predictive maintenancemodel design system.

FIG. 3 is a flow chart depicting steps of another illustrativepredictive maintenance model design process in accordance with aspectsof the present disclosure.

FIG. 4 is a flow chart depicting steps of an illustrative datavisualization and analysis process.

FIG. 5 is a diagram of an illustrative data exploration graphical userinterface (GUI), including a trend plot of a selected phase-dependentaggregate data feature for a selected aircraft for a plurality offlights and maintenance events with associated labeled health periods,over a selected period of time.

FIG. 6 is another illustrative trend plot of a selected phase-dependentaggregate selected data feature for three selected aircraft, asdisplayed by the GUI of FIG. 5.

FIG. 7 is another illustrative trend plot of a selected phase-dependentaggregate data feature for a selected aircraft, including a seasonalbias best-fit curve and maintenance messages, as displayed by the GUI ofFIG. 5.

FIG. 8 is an illustrative flight plot and record table of seven selectedsensor parameters over the course of a selected flight, as displayed bythe GUI of FIG. 5.

FIG. 9 is an illustrative scatter plot of two phase-dependent aggregatedata features for two selected groups of flights, with a linearregression for each group, as displayed by the GUI of FIG. 5.

FIG. 10 is an illustrative density plot of a selected phase-dependentaggregate data feature for two selected groups of flights, as displayedby the GUI of FIG. 5.

FIG. 11 is an illustrative box plot of three selected phase-dependentaggregate data features for two selected groups of flights, as displayedby the GUI of FIG. 5.

FIG. 12 is an illustrative correlation heat map between three selectedphase-dependent aggregate data features for a selected group of flights,as displayed by the GUI of FIG. 5.

FIG. 13 is a schematic diagram of an illustrative data processing systemsuitable for use with aspects of the present disclosure.

DETAILED DESCRIPTION

Various aspects and examples of a predictive maintenance model designsystem including a data visualization module, as well as related systemsand methods, are described below and illustrated in the associateddrawings. Unless otherwise specified, a design system in accordance withthe present teachings, and/or its various components may, but are notrequired to, contain at least one of the structures, components,functionalities, and/or variations described, illustrated, and/orincorporated herein. Furthermore, unless specifically excluded, theprocess steps, structures, components, functionalities, and/orvariations described, illustrated, and/or incorporated herein inconnection with the present teachings may be included in other similardevices and methods, including being interchangeable between disclosedexamples. The following description of various examples is merelyillustrative in nature and is in no way intended to limit thedisclosure, its application, or uses. Additionally, the advantagesprovided by the examples described below are illustrative in nature andnot all examples provide the same advantages or the same degree ofadvantages.

This Detailed Description includes the following sections, which followimmediately below: (1) Overview; (2) Examples, Components, andAlternatives; (3) Illustrative Combinations and Additional Examples; (4)Advantages, Features, and Benefits; and (5) Conclusion. The Examples,Components, and Alternatives section is further divided into subsectionsA through C, each of which is labeled accordingly.

Overview

In general, a predictive maintenance model design system may beconfigured to assist a user in discovering and interpreting data trendsand patterns, designing and training a prediction algorithm, and/orimplementing a generated predictive maintenance model. For example, thedesign system may be a data processing system and/or a software programconfigured to execute a process 110, as shown in FIG. 1, to generate amachine learning model for predicting maintenance requirements of asystem based on historical maintenance and operational data of aplurality of systems by detecting predictive anomalies in ongoingoperational data.

At step 112 process 110 includes integrating operational and maintenancedata, and generating data features. The integration may includereceiving at least two distinct datasets for a plurality of systems suchas a fleet of aircraft, and combining the data to form a singlehistorical dataset. The datasets may be stored in the memory of theprocessing system on which process 110 is executed, may be available ona server for access over a network of some kind, or may be received byany effective means. In some examples, data may be drawn from multipledatabases and/or from disparate sources.

Integrating the data may also include pre-processing or modification toprepare the data for use. The dataset may include numerical valuesorganized as attributes of a plurality of maintenance and/or repairrecords and a plurality of telemetry and/or sensor records for each of aplurality of systems (e.g., aircraft). Raw attribute data from thetelemetry records may be processed to generate one or more operationaldata features. The data features may be an unaltered attribute, may be astatistical function of an attribute, and/or may be an aggregate ofmultiple attributes or records.

At step 114, process 110 includes visualizing and analyzing thehistorical dataset, and receiving a selection of operational datafeatures. The visualization may include displaying a variety of graphs,charts, plots, and tables in a graphical user interface (GUI) along witha plurality of interactive elements. Raw maintenance and operationaldata from the historical dataset, generated data features, and/orresults of analysis of the dataset may be visualized. A user such as anengineering analyst may use the visualizations to identify trends in thesensor data that are indicative of equipment degradation and failure.Facilitating rapid identification of signs of deviation from normaloperation by such an analysis is desirable in order to allow efficientgeneration of useful predictive models.

The interactive elements of the GUI may be configured to allow input ofconstraints on what data is visualized, initiation of desired analysis,and selection of operational data features. For example, the GUI mayinclude selection boxes or buttons, display of contextual information oncursor hover, drill-down from graphs to tables or from complete datasetto data subsets, a refresh trigger, a mathematical function input,and/or any GUI elements known to those skilled in the art of softwaredesign.

At step 116, process 110 includes generating a predictive maintenancemodel based on the selected operational data features. Model generationmay include selection of an appropriate anomaly detection algorithm, andinput of algorithm parameters. In some examples, one or more algorithmtemplates may be presented in the GUI. In some examples, a plurality oflogic block elements may be placeable in an interactive workflowbuilding environment to define an appropriate algorithm and parameters.

Generating the predictive maintenance model may further include trainingand testing the selected algorithm. The algorithm may be trained andevaluated one or more times, and detailed test results displayed in theGUI. A selection of an algorithm configuration exhibiting desiredproperties may be received. The selected algorithm configuration maythen be used to train a final model on the full historical dataset.

At step 118, the process includes implementing the generated predictivemaintenance model. In some examples, the generated model may be preparedfor deployment by software and/or a data processing system separate fromthe predictive maintenance model design system used to execute process110. In such examples, the generated model may be prepared fordeployment as part of a software program, or may be converted to anaccessible format, e.g., including an application programming interface(API).

In some examples, implementing the generated predictive maintenancemodel may include receiving additional operational data. For instance,additional flight data may be recorded by an aircraft fleet and input tothe predictive maintenance model design system running process 110. Thepredictive maintenance model may be applied to the additionaloperational data, and generate alerts for detected anomalies. Based onthe generated alerts, proactive and preventative maintenance action suchas inspection, testing, repair, or replacement of equipment, may betaken by maintenance workers to avoid potential costly and disruptiveunplanned component replacements or other undesirable maintenanceevents.

Process 110 may be repeated to generate additional predictivemaintenance models. For example, a suite of predictive maintenancemodels may be generated for an aircraft fleet. Over time, additionalmodels may be generated or re-generated based on new data and/or toaddress new maintenance challenges.

Aspects of a predictive maintenance model design system or designprocess such as process 110 may be embodied as a computer implementedmethod, computer system, or computer program product. Accordingly,aspects of a predictive maintenance model design system may take theform of an entirely hardware example, an entirely software example(including firmware, resident software, micro-code, and the like), or anexample combining software and hardware aspects, all of which maygenerally be referred to herein as a “circuit,” “module,” or “system.”Furthermore, aspects of the predictive maintenance model design systemmay take the form of a computer program product embodied in acomputer-readable medium (or media) having computer-readable programcode/instructions embodied thereon. The computer-readable program codemay be configured to cause a data processing system to generate apredictive maintenance model.

Any combination of computer-readable media may be utilized.Computer-readable media can be a computer-readable signal medium and/ora computer-readable storage medium. A computer-readable storage mediummay include an electronic, magnetic, optical, electromagnetic, infrared,and/or semiconductor system, apparatus, or device, or any suitablecombination of these. More specific examples of a computer-readablestorage medium may include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, a cloud-based storage service, and/orany suitable combination of these and/or the like. In the context ofthis disclosure, a computer-readable storage medium may include anysuitable non-transitory, tangible medium that can contain or store aprogram for use by or in connection with an instruction executionsystem, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, and/or any suitable combination thereof. Acomputer-readable signal medium may include any computer-readable mediumthat is not a computer-readable storage medium and that is capable ofcommunicating, propagating, or transporting a program for use by or inconnection with an instruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, and/or the like, and/or any suitablecombination of these.

Computer program code for carrying out operations for aspects of apredictive maintenance model design process may be written in one or anycombination of programming languages, including an object-orientedprogramming language such as Java, Smalltalk, C++, Python, and/or thelike, and conventional procedural programming languages, such as C.Mobile apps may be developed using any suitable language, includingthose previously mentioned, as well as Objective-C, Swift, C#, HTML5,and the like.

The program code may execute entirely on a user's computer, partly onthe user's computer, as a stand-alone software package, partly on theuser's computer and partly on a remote computer, or entirely on theremote computer or server. In the latter scenario, the remote computeror server may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), and/or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider). Theremote computer or server may be part of a cloud-based networkarchitecture, such as a cloud computing service or platform. In someexamples, the program code may be executed in a software-as-a-service(SaaS) framework accessed by a file transfer protocol such as secureshell file transfer protocol (SFTP) and/or an internet browser on theuser's computer.

Aspects of the predictive maintenance design system are described belowwith reference to flowchart illustrations and/or block diagrams ofmethods, apparatus, systems, and/or computer program products. Eachblock and/or combination of blocks in a flowchart and/or block diagrammay be implemented by computer program instructions. The computerprogram instructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block(s).In some examples, machine-readable instructions may be programmed onto aprogrammable logic device, such as a field programmable gate array(FPGA).

These computer program instructions can also be stored in acomputer-readable medium that can direct a computer, other programmabledata processing apparatus, and/or other device to function in aparticular manner, such that the instructions stored in thecomputer-readable medium produce an article of manufacture includinginstructions which implement the function/act specified in the flowchartand/or block diagram block(s).

The computer program instructions can also be loaded onto a computer,other programmable data processing apparatus, and/or other device tocause a series of operational steps to be performed on the device toproduce a computer-implemented process such that the instructions whichexecute on the computer or other programmable apparatus provideprocesses for implementing the functions/acts specified in the flowchartand/or block diagram block(s).

Any flowchart and/or block diagram in the drawings is intended toillustrate the architecture, functionality, and/or operation of possibleimplementations of systems, methods, and computer program productsaccording to aspects of the predictive maintenance design system. Inthis regard, each block may represent a module, segment, or portion ofcode, which comprises one or more executable instructions forimplementing the specified logical function(s). In some implementations,the functions noted in the block may occur out of the order noted in thedrawings. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Each block and/or combination of blocks may be implemented byspecial purpose hardware-based systems (or combinations of specialpurpose hardware and computer instructions) that perform the specifiedfunctions or acts.

Examples, Components, and Alternatives

The following sections describe selected aspects of exemplary predictivemaintenance model design systems and data visualization modules as wellas related systems and/or methods. The examples in these sections areintended for illustration and should not be interpreted as limiting theentire scope of the present disclosure. Each section may include one ormore distinct examples, and/or contextual or related information,function, and/or structure.

A. Illustrative Predictive Maintenance Model Design System

As shown in FIGS. 2 and 3, this section describes an illustrative system200, configured to execute a design and implementation process 300.System 200 is an example of a predictive maintenance model designsystem, as described above. Design and implementation process 300 is anexample of process 110, also as described above.

System 200 is configured to assist a user in completing process 300 togenerate and implement machine learning models to detect anomalies inoperational data which are indicative of future maintenance events. Inthe present example, system 200 is designed to generate models for afleet of aircraft, based on recorded flight data. In some examples,system 200 may additionally or alternatively be used for predictivemaintenance with respect to other mechanical systems, such as groundvehicles, ships, manufacturing equipment, industrial appliances, etc.

System 200 may be used to prepare models relating to different aspects(e.g., components or systems) of aircraft, and one set of historicaldata may be used to prepare multiple different models. For example,system 200 may be used to prepare a model for each of multiplesubsystems present in an aircraft, to prepare a model for each class ofsensor recording flight data, or to prepare a model for each failuremode of a particular component.

FIG. 2 is a schematic diagram depicting a remote server 210 and a localcomputer 212 on which design system 200 is executed. The server and/orcomputer may be an example of data processing system 600 as described inExample C, below. The server and local computer communicate, or exchangeinformation, over a network 214 such as the Internet. Multiple modulesof system 200 are run on remote server 210. The modules may comprise,for example, instructions and/or data stored in a memory and executed bya processor. The modules, which may also be referred to as programs orprocesses, include an exploration module 216, a machine learning module218, and an implementation module 220.

Modules 216, 218, 220 may be described as performing steps 310-324 ofprocess 300, in cooperation with a user 228 who performs steps 330-338of the process, as depicted in FIG. 3. Process 300 may also be describedentirely as steps performed by modules 216, 218, 220 including receivinginputs from user 228 as described in steps 330-338.

User 228 interacts with the modules of system 200 on remote server 210through a graphical user interface (GUI) 226 on local computer 212. Theuser is guided through a data exploration and model design process bythe GUI, then a generated model is implemented by remote server 210 onnew operational data to return anomaly alerts. New operational data maybe input and alerts returned on an ongoing basis.

In the present example, GUI 226 is executed by local computer 212. Forinstance, the user may access the GUI through an internet browserinstalled on the local computer, or system 200 may include a client-sideprogram running on the local computer and displaying the GUI. Ingeneral, a user may interface with system 200 in any manner allowingeffective display and input of information.

System 200 may be configured to facilitate exploration of multiplemaintenance events of interest, investigation of multiple trendsconcurrently, design of multiple algorithms, and/or use by multipleusers of local computer 212 through, for example, creation of multipledistinct sessions. Process 300 may be performed or repeatedindependently in each such session, and/or information such as datafeatures, analysis results, or algorithm configurations may beaccessible and/or selectively shared between sessions.

Exploration module 216 receives historical data from a maintenance datasource 222 and a sensor or operational data source 224. In someexamples, the module may receive historical data from a plurality ofmaintenance and/or operational data sources. Typically, flight data arerecorded by airplane sensors and downloaded in large data sets postflight. In some examples, sensor data may additionally or alternativelybe downloaded during a flight using a system such as AircraftCommunications Addressing and Reporting System (ACARS). Separately,airline maintenance teams maintain information logs, recordingmaintenance events, defects, and actions taken, using maintenanceinformation systems in an airline back-office. These two informationsources, flight data and maintenance data, are not integrated. However,in order to perform predictive maintenance analyses, flight datapatterns over time may need to be compared with the occurrence ofmaintenance events.

To address this, exploration module 216 may integrate the datasets foranalysis and visualization. Step 310 of process 300 includes integratingthe historical maintenance and sensor data. The data may be integratedinto a single historical dataset and prepared for display and analysis.For example, the data may be received from other software and may beconverted from an output format of such software to a format appropriateto modules 216, 218 and 220. Preprocessing algorithms may be applied tothe dataset to discretize continuous variables, reduce dimensionality,separate measurements into components, eliminate missing or inaccuratedata, and/or any appropriate modifications. When the dataset includesdata from multiple sources, pre-processing may include merging data,harmonizing formatting, and matching organizational structure.

Exploration module 216 is configured to integrate data saved locally tocomputer 212 (e.g., repair logs saved as spreadsheets and/or in a repairmanagement software database). In some examples, the exploration modulemay also be configured to interface or communicate with externaldatabases or database software to retrieve relevant data. For example,the exploration module may generate SQL queries to request data from anonline database. Such connectivity may facilitate access to complete andup-to-date information. The module may also be configured to acceptinput of data in any anticipated format.

At step 312, exploration module 216 generates data features from theoperational data of the historical dataset, including phase-dependentaggregate data features. Trending of recorded sensor data can bedifficult due to large variations in sensor values that occur as aresult of equipment operational cycles. For example, if an analystsimply aggregates the time series recordings for many aircraft flightsinto one large time series, and plots this data over long periods tolook for a trend over time, the result is typically a very noisydistribution with no significant trends evident.

Digitally recorded sensor data, e.g. temperatures, pressures, electricalcurrents, and actuator positions, from an electro-mechanical system suchas an aircraft or other vehicle, may be discrete samples in a timeseries covering a period of observation. For aircraft, these data arerecorded for each flight. Typical sampling rates are 0.25 Hz to 8 Hz formodern commercial aircraft. The recorded values of these data varysubstantially over the course of an operational cycle, e.g. during theflight of an aircraft. For example, recorded temperatures could vary byhundreds of degrees depending on variables such as altitude (e.g. groundlevel versus cruising altitude), operational mode of the equipment, andany dynamic operational control changes applied to the equipment, eitherautomatically or via an explicit operator, e.g. pilot.

Exploration module 216 avoids this obfuscation by dividing the dataaccording to a plurality of phases of the operational cycle beforeaggregation. For example, sensor data from an aircraft flight may bedivided into phases of taxi-out, take-off, climb, cruise, descent,landing, and taxi-in. A range of aggregating functions may then beseparately applied to the data associated with each phase, to createphase-dependent aggregate features that can be trended over longperiods, e.g. thousands of flights. For instance, the data features maycombine phase-dependence and a value constraint or a differentialcomparison with aggregating statistical functions.

In some examples the module may generate a pre-defined set of datafeatures corresponding to a maintenance event of interest selected bythe user. Such a set may include raw attribute data features and/oraggregate data features. For example, if the user selects an overheatingevent the generated data features may include a temperature sensorreading, an average temperature sensor reading during take-off, adifference in temperature reading between port and starboard sensors,and/or an average difference in temperature reading between port andstarboard sensors during take-off.

At step 330, user 228 may define custom data features. In other words,exploration module 216 may receive a mathematical function or otherlogical ruleset defining a custom data feature from user 228, andgenerate a data feature accordingly. A GUI 226 of design system 200 mayinclude an interface facilitating input of one or more such rulesets.Custom data features may include representations of recorded time-seriessensor data that capture events, transitions, performance metrics, andstates of the system under observation. Any custom defined data featuresmay be ranked, visualized, and communicated to machine learning module218 in the same manner as other data features generated at step 312,such as the pre-defined set of data features.

Exploration module 216 is further configured to categorize maintenanceevents, at step 314 of process 300. More specifically, the module maydivide maintenance events of a selected type into two or moresub-categories based on machine learning analysis of the maintenancedata. For example, topic modelling may be applied to text of maintenancelogs to divide unplanned replacements of a selected component into aplurality of failure types. The maintenance events, sub-categories,and/or other results of the machine learning analysis may be displayedto user 228 through GUI 226.

At step 332, user 228 may designate health timeframes relative torecorded maintenance events. That is, exploration module 216 may receiveone or more time thresholds from the user. For example, user 228 mayindicate that components having undergone replacement are healthy for afirst time period following replacement, and degraded for a second timeperiod preceding replacement. The exploration module may subdivideoperational data into labeled classes or categorizes based on thisreceived designation. In some examples, exploration module 216 mayautomatically use maintenance event sequences to label the sensor datafor the purposes of supervised machine learning, such as classificationmodels for failure prediction.

At step 316, exploration module 216 ranks the importance of some or allof the data features generated in step 312. Importance may be rankedaccording to one or more quantitative measures of influence of a givendata feature on predictive model performance. The ranking may indicaterelative importance of the data features in predicting occurrence of aset of historical maintenance events.

The exploration module may use the designations from step 332 in amachine learning method to perform the ranking. In some examples, theuser may select one or more data features and one or more maintenanceevents to be used in the ranking process. The machine learning methodmay evaluate correlation between data features and maintenance events inan absolute and/or relative sense. For example, the module may iteratesupervised classification, eliminating a data feature at each iterationto generate a relative ranking of feature importance.

At step 334, user 228 may use GUI 226 to explore the sensor andmaintenance data integrated at step 310. The user may also explore thedata features generated at step 312, maintenance event categoriesgenerated at step 314, and/or data feature rankings generated at step316. The user may explore the data in order to identify potential datapatterns, pre-cursor signatures, and/or candidate data features usefulfor creating a predictive maintenance model.

At step 318, exploration module 216 may visualize and analyze data toaid the user's data exploration. The exploration module is configured tovisualize and display both the operational and maintenance data of thehistorical dataset in a manner that enables a user to discover behaviorpatterns in large sets of recorded flight sensor data. Flight sensordata and maintenance event data may be displayed overlaid together inone or more graphs and/or charts to allow the user to identify relevantcorrelations and trends over time. In other words, the explorationmodule automatically combines the flight data features and maintenanceevents into a single time series visualization, enabling visualidentification of important flight data patterns that are associatedwith maintenance problems.

As described above with reference to step 312, trending raw sensor datamay provide limited insight. Instead, exploration module 216 may displaythe generated phase-dependent aggregate data features. The healthtimeframes designated at step 332 may also be displayed relative to eachmaintenance event. Visualizations may be displayed to the user, andconstraints, selections, and other inputs received from the user throughGUI 226.

Exploration module 216 may perform automatic analysis of the historicaldataset as well as additional analysis as selected by the user.Automatic analysis may include standard statistical measures, detectionof potential seasonal bias, and/or any analysis typically relevant tomaintenance prediction. In some examples, the analysis automaticallyperformed may depend on the maintenance event of interest selected bythe user.

GUI 226 may also provide user 228 control over iteration of steps312-316, 330, and 332. For example, user 228 may identify a data trendof interest when exploring sensor and maintenance data in step 334 andperform step 330 again to define a related custom data feature, or maybegin with step 318 to have the exploration module visualize and analyzedata relevant to a data trend of interest before performing step 330 todefine a related custom feature. The user may then trigger generation ofthe defined feature in step 312, and re-ranking of data featureimportance in step 316.

At step 336, user 228 may select one or more key data features. That is,exploration module 216 may receive a selection of one or more datafeatures. In some examples, the user may opt for the exploration moduleto perform an automatic selection of one or more data features. At step320, the exploration module prepares the selected data features and anyrelevant related information, as appropriate. Exploration module 216communicates a set of the data features and labels 233 to machinelearning module 218.

Machine learning module 218 is configured to assist user 228 indefining, training, and evaluating candidate algorithms to arrive at atrained anomaly detection model having desired performance traits. Themachine learning module may be configured to accommodate users ofdifferent levels of expertise and/or allow a user to select a desiredlevel of guidance for a particular project. For example, GUI 226 mayinclude first and second interfaces to the machine learning module.

The first and second interfaces may be designed for beginner andadvanced users, or for simplified and complex design, respectively. Auser may select the first interface due to limited experience, and/or inorder to save time and avoid potential complications such asover-fitting. A user may select the second interface to build and testan algorithm from scratch and/or to create custom algorithmcharacteristics. In some examples, a user may start with a simplealgorithm generated in the first interface, and introduce targetedcomplexity using tools in the second interface.

At step 338, user 228 may select a template and tune algorithmparameters. For example, the user may select an algorithm template withpre-determined parameters through the first interface, or may select analgorithm type and input all relevant parameters through the secondinterface. In either case, the machine learning module 218 may receiveall necessary details of an anomaly detection algorithm configuration.

Appropriate algorithms may include supervised, unsupervised, orsemi-supervised anomaly detection algorithms and techniques such ask-nearest neighbor, support vector machines, Bayesian networks, hiddenMarkov models, or deep learning. Input parameters and/or model settingsmay include tuning parameters such as feature data thresholds andrelative weighting factors, data pre-processing methods such assmoothing, filtering, and normalization, and alert output criteria suchas deviation persistence.

At step 322, the module defines and tests an anomaly detection algorithmbased on the algorithm configuration selected at step 338 and the keydata features selected in step 336. That is, the module trains andvalidates a predictive maintenance model. For example, the prepared datafeatures may be divided into complementary training and validation datasubsets, and the algorithm trained on the training data subset, thentested on the corresponding validation data subset. In some examples,the machine learning module may receive a selection of training andvalidation data sets from the user.

The algorithm may be trained and evaluated one or more times, anddetailed test results reported to user 228 in GUI 226. Based on theevaluation results, the user may repeat step 338 and trigger a repeat ofstep 322 by the machine learning module. The GUI may also providefurther tools for refining the algorithm, such as alternative trainingand testing methods and/or error investigation of individual cases.

In some examples, machine learning module 218 may be configured to trainand evaluate multiple algorithm configurations, either concurrently orsequentially, and report a comparative analysis in addition to or inplace of individual evaluation results. User 228 may repeat steps 338,322 as necessary until arriving at a satisfactory algorithmconfiguration. For example, the user may select for desired propertiessuch as high accuracy or low rate of false positives. The user may thentrigger step 324 of process 300.

Implementation module 220 is configured to apply the final predictivemaintenance model trained by machine learning module 218 to newoperational data, detect anomalies in the data, and generate alertsaccordingly. Machine learning module 218 may communicate a trained model234 to implementation module 220, and the implementation module mayreceive new sensor data 230 from local computer 212. At step 324, themodule runs anomaly detection on the new sensor data. The implementationmodule may then return alerts 232 to the local computer.

Similarly to exploration module 216, implementation module 220 isconfigured to integrate data saved locally to computer 212. In someexamples, the implementation module may also be configured to interfaceor communicate with external databases or database software. Suchconnectivity may facilitate ongoing communication with operationaldatabases for automatic generation of maintenance alerts. For instance,remote server 210 may communicate daily with a database of operationaldata to automatically receive recorded flight data added to the databaseduring maintenance of an aircraft, and issue maintenance alerts beforethe aircraft returns to service.

B. Illustrative Data Visualization Module

As shown in FIGS. 4-12 this section describes an illustrative datavisualization module having a graphical user interface (GUI) 510, and anassociated computer implemented method 400 of data processing anddisplay. The data visualization module may be part of and/or asub-module of an exploration module of a predictive maintenance modeldesign system such as exploration module 216, described above. GUI 510may be an example of GUI 226, as also described above.

The data visualization module is configured to enable a user such as anengineering analyst to identify patterns and pre-cursor signatures inlarge sets of sensor data. Without need for programming expertise, thedata visualization module allows the user to rapidly create long-termtrend plots, perform comparative analysis, select relevant data-subsets,and drill down to individual sensor readings. The data visualizationmodule also facilitates overlay of maintenance data onto an operationaldata timeline, for clear visualization of relationships betweenoperational performance and maintenance outcomes.

A user of the data visualization module may identify patterns for thepurpose of building algorithms to distinguish component degradation andpredict failures in advance. Use of the data visualization module maysubstantially reduce the time required to investigate flight datafeatures and identify features of importance for failure prediction. Insome examples, the data visualization module may be used to identifypatterns relevant to other applications, such as performance evaluationof aircraft models or maintenance facilities. In such examples, the datavisualization module may be configured for standalone use, separate froma design system.

FIG. 4 is a flow chart depicting steps of method 400 of data processingand display. The method may also be described as a method of dataexploration using the data visualization module. The data visualizationmodule receives selections from and displays data plots to the userthrough graphical user interface (GUI) 510, shown in FIGS. 5-12.

Aspects of systems and modules described above may be utilized in themethod steps described below. Where appropriate, reference may be madeto components and systems that may be used in carrying out each step.These references are for illustration, and are not intended to limit thepossible ways of carrying out any particular step of the method. Theflowchart of FIG. 4 may not recite the complete process or all steps ofthe method. Although various steps of method 400 are described below anddepicted in FIG. 4, the steps need not necessarily all be performed, andin some cases may be performed simultaneously or in a different orderthan the order shown.

Step 410 includes receiving a historical dataset of operational andmaintenance data. The dataset may consist of time-labeled historicalmaintenance and sensor data integrated from separate databases, cleaned,and pre-processed for display and analysis, as described in reference tostep 310, above. In the present example the data visualization module isconfigured for use with data from a fleet of aircraft. The operationaldata include flight sensor data, digitally recorded by the fleet ofaircraft and including readings such as temperatures, pressures,electrical currents, and actuator positions. The operational datainclude discrete samples in a plurality of time series, each time seriesrecorded over the course of a flight. The maintenance data includemaintenance logs such as records of scheduled or unplanned componentreplacements, routine upkeep and repair work performed, and inspectionsresults.

Step 412 includes dividing the operational data into phase-dependentsubsets. As discussed above, trending of recorded sensor data can bedifficult due to large variations in sensor values that occur as aresult of equipment operational cycles. In the present example, recordedaircraft flight data varies according to flight phases such as climb,cruise, and descent. To address this, sensor data from each flight isdivided according to flight phases and data is trended for one phase ata time. In other words, the operational data is divided into a pluralityof phase-dependent data subsets, each subset corresponding to one of theflight phases. In examples where the data visualization module isconfigured for use with equipment other than aircraft, the operationaldata may be divided according to other operational phases such as enginewarm-up, full power, and idle.

In the present example, the flight phases are automatically defined astaxi-out, takeoff, climb, cruise, descent, landing, and taxi-in. Theuser may also re-define the flight phases as appropriate to a specificdata analysis goal and/or a type of aircraft. For example, flight phasesfor a fleet including helicopters may further include a hover phase.Step 412 may be repeated as needed when the flight phase definitions arechanged.

Step 414 includes displaying selection tools in a GUI. Morespecifically, selection tools are displayed in GUI 510, as shown in FIG.5. The displayed tools include interactive elements allowing the user tomake selections defining constraints on the data to be visualized andthe types of visualization to be performed. In general, GUI 510 mayinclude any number of selection boxes, dropdown menus, radio buttons,text input boxes, and/or any interactive element effective for input ofdesired constraints. All interactive elements may be displayed in asingle interface view, or the GUI may include contextual menus, optionalwindows with additional tools, and/or any features known to one skilledin the art of interface design.

Step 416 includes receiving selections. The selections are received fromthe user, via GUI 510. Some selections may be required before proceedingwith data visualization in step 432, while other selections may beoptional additional constraints. Step 416 may be repeated may times inmethod 400. The data processing and visualization method may bedescribed as iterative, with repeated selection of constraints anddisplay of data. The user may use insights gained from a visualizationto inform a subsequent selection of constraints, and so on.

The data visualization module or a design system of which the module isa part may allow the user to save a set of selections received at step416, for later use or sharing with other users. For example, the datavisualization module may create distinct user sessions, allowing theuser to work on investigating multiple questions separately, and/ormultiple users to work independently. In some examples, the datavisualization module may support export of a received set of selectionsand/or associated visualizations to another format, or to other dataprocessing software.

In the present example, selection 418 is required before proceeding withstep 432. Selection 418 is of one or more data features. In the presentdescription, the term ‘data feature’ is used to describe a subset ofrecords of the operational data of the historical dataset, extracted,processed, and/or otherwise prepared for use in a machine learningmethod. The data features of selection 418 are defined by selectedconstraints 420-426. More specifically, the data features are defined byconstraint 420 on sensors, constraint 422 on aggregating functions,constraint 424 on flight phase, and constraint 426 on aircraft.

Phase constraint 424 is received via a phase selection menu 520. Theavailable options in menu 520 are the flight phases used to divide theoperational data into phase-dependent data subsets in step 412. In thedepicted example, only a single phase can be selected at a time, and alldata features of selection 418 are restricted to the phase-dependentdata subset corresponding to the selected flight phase. In someexamples, multiple phases may be selected at once using menu 520 or aselection box, and data features for each phase-dependent data subsetcorresponding to a selected flight phase may be included in selection418.

Sensor constraint 420 is received via a sensor parameter selection box512 and a position selection box 516. The term ‘sensor parameter’ isused here to refer to an attribute of the operational data indicating atype of sensor by which values of a record were recorded. An aircraftmay include multiple sensors of the same type, in different positions onthe aircraft. Selection of a sensor parameter and a position mayconstrain the defined data feature to records associated with a singlesensor. The user may select one or more sensor parameters and one ormore positions, to include all sensors defined by possible combinationsof the selected sensor parameters and positions.

In some examples, position selection box 516 may also be used to selectone or more filtering functions to be applied to the data associatedwith the sensor parameters of selection box 512. Use of filtering and/orother statistical functions may reduce observed flight-to-flight noisein displayed features. In the example of FIG. 5, position ‘p7’ isselected with filter ‘rollmed’ and a parameter of 5 is applied. That is,the p7 wheel position of the aircraft is selected and a rolling windowfilter is applied, taking a rolling median over a 5-flight movingwindow.

Aggregating function constraint 422 is received via a function selectionbox 514. In the present example, the available aggregating statisticalfunctions include average, minimum, maximum, standard deviation, median,upper quartile value, and lower quartile value. Similar to theabove-described difficulty in long-term trending due to operationalphase, effective visualization of trends can be hampered by the divisionof the operational data into a plurality of distinct time series, orflights. To avoid this difficulty, the data features of selection 418are aggregate. For each flight, records in the selected phase-dependentdata subset associated with the selected sensor are aggregated accordingto the selected statistical function.

Aircraft constraint 426 is received via a tail number selection box 518.Each aircraft of the fleet of the historical dataset may be designatedby unique identifier. In the present example, each aircraft is assigneda tail number. The user may use box 518 to select one or more tailnumbers for aircraft of interest. The defined data feature may bethereby constrained to records from flights made by the selectedaircraft. The aircraft selected may also be a constraint on displayedmaintenance data, as described further below.

Each possible combination of selected phase, sensor parameters,positions, aggregating functions, and aircraft defines a data feature ofselection 418. Selection 418 may also include a custom data featuredefined by the user. Custom data features may be defined in anothermodule of a design system, and received via an interface opened withfeature button 517. Additionally or alternatively, the datavisualization module may include an interface allowing a user to definea custom data feature. For example, the user may specify an arithmeticcombination of two or more existing data features and/or apply afiltering function or other statistical function.

In the example depicted in FIG. 5, one data feature is selected. The‘landing’ phase, the aircraft assigned identifier ‘Tail_1’, the sensorof type ‘braketemp’ at position ‘p7’ will filter ‘rollmed’, and theaggregating function ‘avg’ or average have been selected to define adata feature named “Tail_1@braketemp@avg@rollmed(p7,5)@landing”. Thedefault name is automatically assigned by the data visualization module.In some examples, the user may opt to replace the automaticallygenerated default name for the data feature with a custom name.

Based on the visualizations of steps 432-454, the user may opt to savethe data feature or features defined by selection 418. Saved datafeatures may be passed to other modules such as a feature rankingmodule, or machine learning module 218, as described above.

Step 416 further includes receiving a selection 428 of maintenance eventtypes, and selection 430 of associated time periods. Along with aircraftconstraint 426, selection 428 may define a subset of maintenance eventsto be displayed, from the maintenance data of the historical dataset.Selection 428 is received via an event selection menu 522. In theexample of FIG. 5, events designated ‘removal unplanned’ are selected.In general, any number of event types may be selected.

The event types available for selection may depend on what events arerecorded in the maintenance logs of the historical dataset, and how theevents are logged. For example, the available types may includeunplanned removal, scheduled removal, diagnostic test, system alert, androutine maintenance. In some examples, a failure mode categorizationmodule or other module of a design system may further divide therecorded maintenance events according to analysis of the maintenancedata. Such division may include assignment to additional event types.For example, unplanned removal events may be divided into electricalfailure, thermal failure, mechanical failure, and unknown failure.

Selection 430 of the event associated time periods is received via ahealthy threshold selection box 524 and a degraded threshold selectionbox 526. Each threshold time defines a time range prior to each event.In other words, the data is divided into multiple time periods accordingto the selected time relationship. More specifically, in the depictedexample the number from box 524 defines a period of days before eachevent, prior to which operational data is assumed to reflect healthyoperation. That is, a healthy period from the most recently precedingevent to the selected number of days prior to the respective event isdefined. The number from box 526 defines a period of days before anevent during which operational data is assumed to reflect degradedcomponents or unhealthy operation. That is, a degraded period of theselected number of days up to the event is defined. Any time not withineither period for any selected maintenance event is designated as havingan uncertain status.

The depicted time periods are labeled for use with unplanned removalevents, to assign healthy and degraded labels based on an assumption ofcomponent degradation leading to the unplanned removal. However, theuser may use boxes 524 and 526 to define any informative time periodsprior to a selected event type. For example, a user investigatingeffects of routine maintenance on aircraft performance might designatetime periods corresponding to theorized optimal maintenance intervals tovisualize operational data from flights outside such intervals.

Step 432 of method 400 includes displaying values for eachphase-dependent aggregate data feature in a trend plot. That is, valuesare calculated from operational data and displayed as points of a plotor graph. A calculated value is achieved by application of the selectedaggregating statistical function to data from each selected sensor dataparameter for the selected systems and the selected operational phase.

In the example depicted in FIG. 5, GUI 510 includes a trend plot 530 ofselected data feature Tail_1@braketemp@avg@rollmed(p7,5)@landing. Trendplot 530 is a scatter plot, with a horizontal time axis 532 and avertical feature value axis 534. The user may select a date domain fortime axis 532 using two date selection boxes 536, and either allow anautomatic value range selection for value axis 534 or opt to impose aselected value range using checkbox 538.

Each point 540 of trend plot 530 corresponds to a flight made byaircraft Tail_1. The point is plotted at the date of the flight, and thevalue of the point is the average calculated from all filtered values ofbrake temperature readings by a temperature sensor at a p7 wheelposition, during the landing phase of the flight. The filtered value iscalculated from the brake temperature readings by averaging readingsover a five-flight moving window.

In general, for every displayed trend plot, each scatterplot pointindicates a value calculated from a series of sensor readings recordedon a flight. More specifically, the point indicates a value calculatedfrom sensor readings recorded during one phase of a flight. Suchphase-dependent aggregate display of sensor data allows much clearerdisplay of high-level trends over the course of months and years,reducing the noise associated with variation within individual flights.

By default, each data feature is displayed on a separate trend plot. Inexamples where a user selects multiple data features, additional trendplots may be displayed below trend plot 530 in GUI 510. Each trend plotmay be titled with the name of the plotted data feature, as shown fortrend plot 530 in FIG. 5. The user may also opt to overlap selected datafeatures, using overlap checkboxes 542.

In the present example, overlap is allowed for data features sharing allconstraints apart from aircraft and/or position. An overlap checkbox 542is therefore associated with each of tail number selection box 518 andposition selection box 516. In some examples, overlap may be permittedfor any desired data features, and an overlap checkbox may also beassociated with each of sensor parameter selection box 512, functionselection box 514 and phase selection menu 520.

An example of a trend plot 544 with overlapped aircraft is depicted inFIG. 6. Three tail numbers are selected, and plot points 540 includeflights taken by all three aircraft. Operational data of a first of theselected aircraft is visually distinct from operational data of a secondof the selected aircraft, both of which are distinct from operationaldata of the third of the aircraft. In some examples, the plot points maybe color-coded to indicate which aircraft the respective flightassociated with. In the present example, points 540 are plotted withthree different symbols to represent the three different aircraft. Thetitle of trend plot 544 is named by default to reflect all constraintsshared by the three plotted data features, specifically the sensorparameter, aggregating function, sensor position, and flight phase.

Maintenance events are also displayed in each trend plot. At step 434,the method includes displaying maintenance events of the selected typeassociated with the respective aircraft. The module may displaymaintenance data by overlaying the time-labeled maintenance data on thesame timeline as the displayed operational data. For each trend plotdisplayed in step 432, maintenance events of the types received inselection 428 associated with the aircraft selected in the definition ofthe respective data feature are displayed. More specifically, a labeledvertical line or event line is plotted at the date of the event.

In the example depicted in FIG. 5, only one maintenance event of theselected unplanned removal type occurred in the selected date domain foraircraft Tail_1. Accordingly, a single dashed event line 546 is plottedon trend plot 530, with the label ‘removal unplanned’. In the exampledepicted in FIG. 6, three maintenance events occurred in the selecteddate domain for the three selected aircraft, and three event lines 546are plotted in trend plot 544. In an example depicted in FIG. 7, bothunplanned removals and scheduled removals are selected and a trend plot548 includes three event lines 546 labeled as ‘removal scheduled’ andone event line labeled as ‘removal unplanned’.

GUI 510 provides additional, detailed information for individualmaintenance events, as request by a user. For example, hovering a cursorover maintenance event lines on the trend plot may trigger a popuptooltip, contextual menu, or highlight label with information such asmaintenance log text or message text. In FIG. 5, a highlight label forthe unplanned removal event of event line 546 displays information onthe maintenance event, including that the carbon brake at wheel positionp7 was replaced because the previous brake was worn to the limit.

Step 436 includes visually indicating a division into the selected timeperiods, for each displayed maintenance event. ** A division of eachtrend plot into the time periods of selection 430 as defined by thethreshold values of selection boxes 524 and 526 is indicated. In someexamples, the time periods may be represented as color-coded regions ofthe trend plot. In the examples depicted in FIGS. 5 and 6, the timeperiods are shown as patterned regions of trend plots 530 and 544. A key550 is provided in GUI 510 (FIG. 5) to the defined ‘Assumed Degraded’and ‘Assumed Healthy’ periods.

Such display of maintenance and operational data on the same timeline isof substantial benefit in discovering trends in operational data thatare correlated with maintenance events and therefore useful inprediction of such events. For instance, in the example of FIG. 5, asignificant trend can be observed in the displayed brake temperature. Inthe degraded time period leading up to the unplanned replacement of thebrake, recorded brake temperatures rise. Immediately following thereplacement, the temperature return to levels at or below those of thepreceding healthy period. Increase in this brake temperature maytherefore be a pre-cursor signature to brake failure, and the displayeddata feature may be useful for generation of a predictive maintenancemodel.

The data visualization module further supports display of othermaintenance data, including maintenance messages from the aircraftonboard computer, and fault messages from the Engine Indicating and CrewAlerting System (EICAS). Such messages are indicated by a box 554 and anoval 556 on the trend plot, as depicted on trend plot 548 in FIG. 7. Themessage may provide additional context to a user for understandingobserved data patterns. In some cases, a pattern may be observed inaircraft messages leading up to a maintenance event.

After step 436, the next step in method 400 may be triggered by theuser, in GUI 510. The user may return to the selection tools to alterthe selections, and the method may return to step 416. The user may optto evaluate seasonality of the selected data features, using seasonalbias analysis checkbox 558 or seasonal bias removal checkbox 559, inwhich case the method may proceed with step 438. The user may opt todrill down to flight level data by selecting button 560, in which casethe method may proceed with step 446. The user may opt to generateadditional plots by selecting button 562 or 564, in which case method400 may proceed with step 450 or 452, respectively. The user may iterateany or all of these optional branches of method 400 as desired, byinteraction with the relevant elements of GUI 510.

Step 438 includes calculating a best-fit periodic curve. As noted above,step 438 may be performed as an analysis for seasonal bias or variationpresent in the data, and may be triggered by checkbox 558 or 559 in GUI510. The best-fit calculation may be performed for each selected datafeature, within the selected date domain. More specifically, the datavisualization module performs a sinusoidal regression for each datafeature time series, and temporarily stores the fit results.

At decision 440, the module evaluates the parameters of the stored fitresults, comparing the parameters to pre-determined thresholds forseasonal bias. For example, a period of a calculated best-fit curve maybe compared to an annual period of a year or 365 days. If the period iswithin a pre-determined threshold, the data qualifies as having aseasonal bias. For example, if the calculated period is within 40 daysof 365 days, the data is considered to have seasonal bias. In someexamples, the pre-determined thresholds may be customizable by the user.

If the result of the evaluation at 440 is that no seasonal bias ispresent, method 400 may return to step 432. Trend plots of the selecteddata features may be displayed without alteration. In some examples, anotification may be displayed to the user indicating the result of theseasonal bias analysis. If one or more of the data features does exhibitseasonal bias, the method proceeds with step 442 or step 444 accordingto the GUI checkbox selected by the user. Each step is performed forthose data features exhibiting seasonal bias. Any data features with noseasonal bias may be displayed without alteration.

For a selection of seasonal bias analysis checkbox 558, step 444includes displaying each calculated curve on the respective trend plot.An illustrative seasonal curve 568 is displayed in trend plot 548 ofFIG. 7. As indicated in the title of the trend plot, and date selectionboxes 536, trend plot 548 depicts average temperature at a first leftcabin air compressor outlet, during the climb phase of flight, forflights between January and July of one year. Curve 568 exhibitsapproximately half periodicity in the depicted time, consistent withseasonal variation. The best-fit curve is overlaid with plot points 540,allowing the user to visually assess curve fit, the strength or extentof seasonal variation, and/or data patterns distinct from the seasonalvariation.

Seasonal bias is common in recorded aircraft sensor parameters such astemperatures, pressures, etc. measured during lower altitude phases offlight. Periodicity of a data feature due to seasonal bias may result ingeneration of false positive alerts by a predictive algorithm. However,a seasonally biased data feature may still have predictive value,particularly if the bias can be removed.

For a selection of seasonal bias removal checkbox 559, step 442 includesmodifying each data feature to compensate for the seasonal variation.The data visualization module may use the stored fit results to alterthe data feature values. More specifically, the sinusoidal regressioncurve values may be subtracted from the data feature values, and analtered data feature stored. Method 400 returns to step 432, and thealtered data features are used in place of the original data features inthe displayed trend plots. In some examples, the trend plots of altereddata features may be labeled as modified to compensate for calculatedseasonal bias, or a notification may be displayed to the user indicatedwhich data features were altered in the seasonal bias analysis.

The data visualization module is configured to visualize aircraft flightsensor data at multiple levels. GUI 510 allows rapid drill-down from thelong-term trending analysis of aggregate data parameters over thousandsof historical flights to exploration of raw sensor data in individualflights. Step 446 includes receiving a selection of a flight. In thepresent example, the user uses button 560 to activate flightvisualization and then clicks a plot point 540 in trend plot 530 toselect an individual flight.

Step 448 includes displaying readings from the flight for each selectedsensor, in a separate plot. That is, flight data is displayed on aseparate timeline from the trend plots. In the present example, once theuser has selected a flight, GUI 510 opens a new window as depicted inFIG. 8. The window includes a flight plot 570, a sensor selection box572, and a sensor parameter record table 574. In the depicted example,flight plot 570 is a line graph with a horizontal time axis 576. Thetime domain is the duration of the flight, and the value rangeautomatically adjusts to accommodate maximum values to be displayed. Aline 580 is graphed for each selected sensor. In the depicted example,seven data parameters are graphed, including three with zero value overa majority of the flight.

Initially, flight plot 570 may graph data recorded by the sensors ofconstraint 420 in selection 418 of step 416. The user may subsequentlyadd and/or subtract sensors as desired using sensor selection box 572.In some examples, flight phases may be visually indicated on flight plot570. For example, the graph lines may be color-coded according to flightphase, or vertical lines dividing the phase time ranges may bedisplayed.

More detailed data is also available to the user in record table 574 anda contextual highlight label 582. As the user hovers a cursor overflight plot 570, points on each graph line 580 at the corresponding timeare highlighted and label 582 is displayed with values for eachhighlighted point. Only a portion of record table 574 is depicted inFIG. 8, but the table includes a row for each record made over thecourse of the selected flight, and a column for each selected sensor.The user may thereby access raw operational data from the historicaldataset, in a limited and manageable manner.

In addition to long-term trending and individual flight level data, theuser may opt to view comparative plots for one or more aircraft, or oneor more groups of flights. Four illustrative plots are depicted in FIGS.9-12, including a scatter plot with linear regression, a density plot, abox plot, and a heat map. In general, GUI 510 and the data visualizationmodule may be configured to provide any visualizations appropriate tothe intended application. For example, the plots of FIGS. 9-12facilitate investigation of relationships between data features forbetter predictive model design. Options for plots, graphs, charts, orother visualizations offering similar insights relevant to other typesof data analysis may be provided in the GUI.

Step 450 includes receiving a selection of one or more aircraft. In thepresent example, the user may make the selection using tail numberselection box 518. The user may then trigger step 454 with visualizationbutton 562, as shown in FIG. 5. Alternatively, step 452 includesreceiving a selection of two groups of flights. To select each group,the user may click one of group definition buttons 566 and then clickand drag in trend plot 530 to select a plurality of plot points 540 andthereby select a plurality of associated flights. A number of selectedflights associated with each group definition button 566 is displayed atthe top right corner of the button. Once each of the two groups isselected, the user may trigger step 454 with visualization button 564.

Selection of flight groups may be particularly helpful for comparingdata from different time regions as defined in selection 430 anddisplayed in step 436. For example, a first flight group may be selectedfrom a time region labeled as healthy and second flight group may beselected from a time region labeled as degraded. Comparative plots maythen allow the user to identify statistical differences between healthyand degraded sensor data. If there is a substantial difference betweenthe selected groups, then the data feature may be a good candidate for apredictive alerting algorithm.

Step 454 includes displaying one or more of a scatter plot with a linearregression, a density plot, a box plot, and a heat map for the selecteddata features. The data features displayed in the plot are thoseselected at 418 of step 416. The plots to be displayed maypre-determined or may be selectable by the user. In the present example,all four plot types are displayed automatically. The same plots aredisplayed either for the aircraft selected in step 450, or the groups offlights selected in step 452. The illustrative plots of FIGS. 9-12 arefor two selected groups of flights. It may be understood that the plotdepictions and the descriptions below similarly apply to plots for oneor more selected aircraft.

FIG. 9 depicts a scatter plot 584 with a horizontal axis of values for afirst data feature and a vertical axis of values for a second datafeature. In the present example the first data feature is a standarddeviation of inner wing flaps sensor readings and the second datafeature is a standard deviation of outer wing flaps sensor readings. Thedata visualization module may display a corresponding plot for eachcombination of selected data features. Points 587 are plotted in scatterplot 584 for each of a first group of flights and a second group offlights. For each group, a linear regression is performed and a best-fitline 589 is displayed in scatter plot 584. On cursor hover, a highlightlabel 590 displays specific values for each plot point 587.

A plot key 592 includes a calculated p-value for each best-fit line, anda correlation score for each group. The correlation score may help theuser to evaluate independence of data features, and identify anomalousdata. For instance, if two variables are highly correlated then one maybe omitted without significant loss of information, and thereby avoidissues with predictive models that do not perform well with redundantdata features. For another instance, anomalous data may appear asregions of non-correlation, which may be indicative of a predictivesignature.

FIG. 10 depicts a probability density plot 583 for the two selectedgroups of flights, and one of the selected data features, as identifiedin the plot title. The data visualization module may display acorresponding plot for each selected data feature. A densitydistribution 585 is displayed for a first of the selected flight groupsand a density distribution 586 is displayed for a second of the selectedflight groups, as identified by a plot key 592. On cursor hover, ahighlight label 590 displays specific density numbers for each flightgroup at the respective value.

FIG. 11 depicts a box plot 591 for the two selected groups of flights,and all the selected data features. For each data feature, a pair ofboxes and whiskers 588 is displayed, one for each flight group asidentified by a plot key 592. The boxes are vertically oriented, with avertical value axis and data features along a horizontal axis of boxplot 591. On cursor hover, a highlight label 590 displays details ofoutliers for a respective box and whiskers 588.

FIG. 12 depicts a heat map 593 of correlation between all of theselected data features, for one of the selected flight groups, asindicated by the plot title. The data visualization module may display acorresponding plot for the other selected flight group. Heat map 593includes a cell 596 for each pair of the selected data features.Correlation between the pair of data features is evaluated and aquantitative result of the evaluation is displayed as a correlationnumber. A grey value or pattern density is also displayed in the cellmatching the correlation number, according to a heat scale 598. Oncursor hover, a highlight label 590 identifies the pair of data featurescorresponding to the cell and displays a more specific value for thecorrelation number.

Such comparative analyses as facilitated by plots 584, 585, 591, and 593may allow the user to determine which data features provide independentinformation useful in developing predictive alerting algorithms. Thedata visualization module may further display any charts, plots, heatmaps, and/or graphs useful for identification of trends or patterns indata on either a long-term or flight level timescale.

From step 454 the user may return to the selection tools of GUI 510, asdepicted in FIG. 5, to repeat step 416 and any desired subsequent steps.In some examples, the user may export a portion of the historicaldataset corresponding to selections received at step 416, and/orvisualizations generated in steps 432-454. After repeating steps ofmethod 400 as needed to identify one or more data patterns or pre-cursorsignatures, the user may opt to save the data feature or featuresdefined by selection 418 for use in predictive model training.

C. Illustrative Data Processing System

As shown in FIG. 13 this example describes a data processing system 600(also referred to as a computer, computing system, and/or computersystem) in accordance with aspects of the present disclosure. In thisexample, data processing system 600 is an illustrative data processingsystem suitable for implementing aspects of the predictive maintenancemodel generation and data visualization methods described above. Morespecifically, in some examples, devices that are examples of dataprocessing systems (e.g., servers, tablets, personal computers) may runone or more modules of a predictive maintenance model design system toexecute the methods described above.

In this illustrative example, data processing system 600 includes asystem bus 602 (also referred to as communications framework). Systembus 602 may provide communications between a processor unit 604 (alsoreferred to as a processor or processors), a memory 606, a persistentstorage 608, a communications unit 610, an input/output (I/O) unit 612,and/or a display 614.

Processor unit 604 serves to run instructions that may be loaded intomemory 606. Processor unit 604 may comprise a number of processors, amulti-processor core, and/or a particular type of processor orprocessors (e.g., a central processing unit (CPU), graphics processingunit (GPU), etc.), depending on the particular implementation. Further,processor unit 604 may be implemented using a number of heterogeneousprocessor systems in which a main processor is present with secondaryprocessors on a single chip.

Memory 606 and persistent storage 608 are examples of storage devices616. A storage device may include any suitable hardware capable ofstoring information (e.g., digital information), such as data, programcode in functional form, and/or other suitable information, either on atemporary basis or a permanent basis. Storage devices 616 also may bereferred to as computer-readable storage devices or computer-readablemedia.

Persistent storage 608 may contain one or more components or devices.For example, persistent storage 608 may include one or more devices suchas a magnetic disk drive (also referred to as a hard disk drive or HDD),solid state disk (SSD), an optical disk drive such as a compact disk ROMdevice (CD-ROM), flash memory card, memory stick, and/or the like, orany combination of these. One or more of these devices may be removableand/or portable, e.g., a removable hard drive.

Input/output (I/O) unit 612 allows for input and output of data withother devices that may be connected to data processing system 600 (i.e.,input devices and output devices). For example, an input device mayinclude one or more pointing and/or information-input devices such as akeyboard, a mouse, touch screen, microphone, digital camera, and/or thelike. These and other input devices may connect to processor unit 604through system bus 602 via interface port(s) such as a serial portand/or a universal serial bus (USB).

Output devices may use some of the same types of ports, and in somecases the same actual ports, as the input device(s). For example, a USBport may be used to provide input to data processing system 600 and tooutput information from data processing system 600 to an output device.Some output devices (e.g., monitors, speakers, and printers, amongothers) may require special adapters. Display 614 may include anysuitable human-machine interface or other mechanism configured todisplay information to a user, e.g., a CRT, LED, or LCD monitor orscreen, etc.

Communications unit 610 refers to any suitable hardware and/or softwareemployed to provide for communications with other data processingsystems or devices. While communication unit 610 is shown inside dataprocessing system 600, it may in some examples be at least partiallyexternal to data processing system 600. Communications unit 610 mayinclude internal and external technologies, e.g., modems, ISDN adapters,and/or wired and wireless Ethernet cards, hubs, routers, etc. Dataprocessing system 600 may operate in a networked environment, usinglogical connections to one or more remote computers.

Instructions for the operating system, applications, and/or programs maybe located in storage devices 616, which are in communication withprocessor unit 604 through system bus 602. In these illustrativeexamples, the instructions are in a functional form in persistentstorage 608. These instructions may be loaded into memory 606 forexecution by processor unit 604. Processes of one or more examples ofthe present disclosure may be performed by processor unit 604 usingcomputer-implemented instructions, which may be located in a memory,such as memory 606.

These instructions are referred to as program instructions, programcode, computer usable program code, or computer-readable program codeexecuted by a processor in processor unit 604. The program code in thedifferent examples may be embodied on different physical orcomputer-readable storage media, such as memory 606 or persistentstorage 608. Program code 618 may be located in a functional form oncomputer-readable media 620 that is selectively removable and may beloaded onto or transferred to data processing system 600 for executionby processor unit 604. Program code 618 and computer-readable media 620form computer program product 622 in these examples. In one example,computer-readable media 620 may comprise computer-readable storage media624 or computer-readable signal media 626.

Computer-readable storage media 624 may include, for example, an opticalor magnetic disk that is inserted or placed into a drive or other devicethat is part of persistent storage 608 for transfer onto a storagedevice, such as a hard drive, that is part of persistent storage 608.Computer-readable storage media 624 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory, that is connected to data processing system 600. In someinstances, computer-readable storage media 624 may not be removable fromdata processing system 600.

In these examples, computer-readable storage media 624 is anon-transitory, physical or tangible storage device used to storeprogram code 618 rather than a medium that propagates or transmitsprogram code 618. Computer-readable storage media 624 is also referredto as a computer-readable tangible storage device or a computer-readablephysical storage device. In other words, computer-readable storage media624 is media that can be touched by a person.

Alternatively, program code 618 may be transferred to data processingsystem 600, e.g., remotely over a network, using computer-readablesignal media 626. Computer-readable signal media 626 may be, forexample, a propagated data signal containing program code 618. Forexample, computer-readable signal media 626 may be an electromagneticsignal, an optical signal, and/or any other suitable type of signal.These signals may be transmitted over communications links, such aswireless communications links, optical fiber cable, coaxial cable, awire, and/or any other suitable type of communications link. In otherwords, the communications link and/or the connection may be physical orwireless in the illustrative examples.

In some illustrative examples, program code 618 may be downloaded over anetwork to persistent storage 608 from another device or data processingsystem through computer-readable signal media 626 for use within dataprocessing system 600. For instance, program code stored in anon-transitory computer-readable storage medium in a server dataprocessing system may be downloaded over a network from the server todata processing system 600. The computer providing program code 618 maybe a server computer, a client computer, or some other device capable ofstoring and transmitting program code 618.

The different components illustrated for data processing system 600 arenot meant to provide architectural limitations to the manner in whichdifferent examples may be implemented. One or more examples of thepresent disclosure may be implemented in a data processing system thatincludes fewer components or includes components in addition to and/orin place of those illustrated for computer 600. Other components shownin FIG. 13 can be varied from the examples depicted. Different examplesmay be implemented using any hardware device or system capable ofrunning program code.

Illustrative Combinations and Additional Examples

This section describes additional aspects and features of predictivemaintenance model design systems, computer implemented methods ofpredictive maintenance design, and computer programs product forgenerating predictive maintenance models, presented without limitationas a series of paragraphs, some or all of which may be alphanumericallydesignated for clarity and efficiency. Each of these paragraphs can becombined with one or more other paragraphs, and/or with disclosure fromelsewhere in this application, including the materials incorporated byreference in the Cross-References, in any suitable manner. Some of theparagraphs below expressly refer to and further limit other paragraphs,providing without limitation examples of some of the suitablecombinations.

A0. A data processing system for generating predictive maintenancemodels, comprising:

one or more processors,

a memory including one or more digital storage devices, and

a plurality of instructions stored in the memory and executable by theone or more processors to:

receive a historical dataset relating to each system of a plurality ofsystems, the historical dataset including time-labeled maintenance andoperational data,

receive a first selection of a first operational data feature and afirst system,

display operational data associated with the first operational datafeature and the first system on a timeline in a graphical userinterface,

display maintenance data associated with the first system on thetimeline,

receive a second selection of a second operational data feature, and

generate a predictive maintenance model, using the second operationaldata feature according to a machine learning method.

A1. The data processing system of A0, wherein the timeline is a timeaxis of a scatterplot, and the operational data associated with thefirst operational data feature and the first system are displayed aspoints of the scatterplot.

A2. The data processing system of A1, wherein the plurality of systemsare a fleet of aircraft, and each point of the scatterplot indicates avalue calculated from a series of sensor readings recorded on a flight.

A3. The data processing system of any of A0-A2, wherein the plurality ofsystems have a shared plurality of operational phases, and the pluralityof instructions are further executable by the one or more processors to:

divide the operational data into a plurality of phase-dependent datasubsets corresponding to the phases of the shared plurality ofoperational phases.

A4. The data processing system of A3, wherein the first selectionincludes an operational phase, and the displayed operational data is ina phase-dependent data subset corresponding to the operational phase.

A5. The data processing system of A3 or A4, wherein the plurality ofsystems are a fleet of aircraft, and the shared plurality of operationalphases are phases of flight.

A6. The data processing system of A5, wherein the shared plurality ofoperational phases include one or more of taxi-out, takeoff, climb,cruise, descent, landing, and taxi-in.

A7. The data processing system of A5 or A6, wherein the operational dataof the historical dataset includes a plurality of series of sensorreadings, each series of sensor readings having been recorded during aflight of an aircraft of the fleet of aircraft, and the plurality ofinstructions are further executable by the one or more processors to:

receive a selection of a flight, and

display the series of sensor readings recorded during the selectedflight, on a separate timeline.

A8. The data processing system of any of A3-A7, wherein the firstoperational data feature is an aggregate data feature, calculatedaccording to an aggregating statistical function on one of thephase-dependent data subsets of the operational data.

A9. The data processing system of A8, wherein displaying the operationaldata associated with the first operational data feature includesdisplaying a calculated value of the aggregating statistical functionfor each of a plurality of series of sensor readings.

A10. The data processing system of A8 or A9, wherein receiving the firstselection of a data feature includes receiving a selection of a sensordata parameter, a position, and the aggregating statistical function,and the aggregate data feature is calculated from operational dataassociated with the sensor data parameter and the position according tothe selected aggregating statistical function.

A11. The data processing system of any of A3-A10, wherein the firstselection includes a plurality of aggregate operational data featurescalculated according to one or more aggregating statistical functions,and the plurality of instructions are further executable by the one ormore processors to:

receive a selection of an operational phase, and

calculate each aggregate data feature for the selected operationalphase,

wherein displaying the operational data associated with the firstoperational data features includes displaying calculated values of theaggregate data features.

A12. The data processing system of any of A0-A11, wherein the firstoperational data feature is an aggregate data feature, calculatedaccording to an aggregating statistical function.

A13. The data processing system of any of A0-A12, wherein the pluralityof instructions are further executable by the one or more processors to:

analyze seasonal variation in the operational data associated with thefirst operational data feature and the first system,

calculate a best-fit seasonal bias curve, and

display the calculated curve on the timeline of the graphical userinterface.

A14. The data processing system of A13, wherein the plurality ofinstructions are further executable by the one or more processors to:

modify the operational data associated with the first operational datafeature and the first system to compensate for the analyzed seasonalvariation, prior to displaying the operational data on the timeline ofthe graphical user interface.

A15. The data processing system of any of A0-A14, wherein the pluralityof instructions are further executable by the one or more processors to:

calculate a best-fit periodic curve for the operational data associatedwith the first operational data feature and the first system,

compare a period of the calculated curve to a year, and

when the period is within a pre-determined threshold from a year,display the calculated curve on the timeline of the graphical userinterface.

A16. The data processing system of A15, wherein the plurality ofinstructions are further executable by the one or more processors to:

modify the operational data associated with the first operational datafeature and the first system to compensate for the calculated curve,prior to displaying the operational data on the timeline of thegraphical user interface.

A17. The data processing system of any of A0-A16, wherein the firstselection includes a plurality of operational data features, and theplurality of instructions are further executable by the one or moreprocessors to:

display operational data associated with each of the plurality ofoperational data features on the timeline, the operational dataassociated with each operational data feature of the plurality ofoperational data features being visually distinct from operational dataassociated with the other operational data features of the plurality ofoperational data features.

A18. The data processing system of A17, wherein the plurality ofinstructions are further executable by the one or more processors to:

evaluate correlation between each pair of the plurality of operationaldata features, and

display a quantitative result of each evaluation in the graphical userinterface.

A19. The data processing system of any of A0-A18, wherein the firstselection includes a plurality of systems, and the plurality ofinstructions are further executable by the one or more processors to:

display operational data associated with each system of the plurality ofsystems on the timeline, the operational data associated with eachsystem of the plurality of systems being visually distinct fromoperational data associated with the other systems of the plurality ofsystems.

A20. The data processing system of any of A0-A19, wherein displayingmaintenance data associated with the first system includes displaying amaintenance event for the first system, and the plurality ofinstructions are further executable by the one or more processors to:

visually indicate a division of the displayed operational data intomultiple time periods according to a selected time relationship to thedisplayed maintenance event.

A21. The data processing system of A20, wherein the multiple timeperiods include a first period of time more than a first threshold timeprior to the displayed maintenance event, and a second period of timeless than a second threshold time prior to the displayed maintenanceevent, and the plurality of instructions are further executable by theone or more processors to:

receive a selection of the first and second threshold times.

A22. The data processing system of any of A0-A21, wherein the pluralityof instructions are further executable by the one or more processors to:

receive a selection of a pair of operational data features, systems, orpluralities of series of sensor readings, and

display a comparison of the selected pair in one or more of a densityplot, a heat map, a box plot, and a scatter plot including a linearregression.

A23. The data processing system of any of A0-A22, wherein the pluralityof instructions are further executable by the one or more processors to:

receive a selection of two groups of series of sensor readings, and

display values of the first data feature in one or more of a densityplot, a box plot, and a scatter plot including a linear regression.

A24. The data processing system of A23, wherein the first selectionincludes a plurality of operational data features, and the plurality ofinstructions are further executable by the one or more processors to:

evaluate correlation between each pair of the plurality of operationaldata features for each group, and

display a quantitative result of each evaluation.

A25. The data processing system of A23 or A24, wherein the firstselection includes a plurality of operational data features, and theplurality of instructions are further executable by the one or moreprocessors to:

visually indicate a division of the displayed operational data intomultiple time periods according to a selected time relationship to adisplayed maintenance event, the multiple time periods including a firstperiod of time more than a first threshold time prior to the displayedmaintenance event, and a second period of time less than a secondthreshold time prior to the displayed maintenance event, and

wherein a first group of the two groups of series of sensor readings isselected from the first time period, and a second group of the twogroups of series of sensor readings is selected from the second timeperiod.

B0. A data processing system for generating predictive maintenancemodels, comprising:

one or more processors;

a memory including one or more digital storage devices; and

a plurality of instructions stored in the memory and executable by theone or more processors to:

receive a historical dataset relating to each system of a plurality ofsystems, the historical dataset including time-labeled maintenance andoperational data, and the plurality of systems having a shared pluralityof operational phases,

divide the operational data into a plurality of subsets corresponding tothe shared plurality of operational phases,

receive a first selection of operational data features, systems of theplurality of systems, and an operational phase of the shared pluralityof operational phases,

display the operational data subset corresponding to the selectedoperational phase in a graphical user interface, and

receive a second selection of operational data features from the user,and

generate a predictive maintenance model, using the second selection ofoperational data features according to a machine learning method.

C0. A computer implemented method of generating a predictive maintenancemodel, comprising:

receiving a historical dataset relating to each system of a plurality ofsystems, the historical dataset including time-labeled maintenance andoperational data,

receiving a first selection of a first operational data feature and afirst system,

displaying operational data associated with the first operational datafeature and the first system on a timeline in a graphical userinterface,

displaying maintenance data associated with the first system on thetimeline,

receiving a second selection of a second operational data feature, and

generating a predictive maintenance model, using the second operationaldata feature according to a machine learning method.

D0. A computer program product for generating predictive maintenancemodels, the computer program product comprising:

a non-transitory computer-readable storage medium havingcomputer-readable program code embodied in the storage medium, thecomputer-readable program code configured to cause a data processingsystem to generate a predictive maintenance model, the computer-readableprogram code comprising:

at least one instruction to receive a historical dataset relating toeach system of a plurality of systems, the historical dataset includingtime-labeled maintenance and operational data,

at least one instruction to receive a first selection of a firstoperational data feature and a first system,

at least one instruction to display operational data associated with thefirst operational data feature and the first system on a timeline in agraphical user interface,

at least one instruction to display maintenance data associated with thefirst system on the timeline,

at least one instruction to receive a second selection of a secondoperational data feature, and

at least one instruction to generate a predictive maintenance model,using the second operational data feature according to a machinelearning method.

E0. A data processing system for analyzing data relating to aircraftoperation and maintenance, comprising:

a processing system;

a memory including a digital storage device; and

a plurality of instructions stored in the memory and executable by theprocessing system to:

receive a historical dataset relating to a plurality of aircraft, thehistorical dataset including time-labeled maintenance and operationaldata,

receive a first selection of operational data features and a selectionof aircraft from a user,

display the operational data for the first selection of operational datafeatures and the selected aircraft on a timeline in a graphical userinterface, and

display the maintenance data for the selected aircraft on the timeline.

E1. The data processing system of E0, wherein the plurality ofinstructions are further executable by the processing system to:

receive a second selection of operational data features from the user,and

generate a predictive maintenance model, using the second selection ofoperational data features according to a machine learning method.

E2. The data processing system of E0 or A1, wherein the operational datainclude a series of sensor readings from a first piece of equipment foreach aircraft, and the plurality of instructions are further executableby the one or more processors to:

divide each series of sensor readings into a plurality of time periodscorresponding to a plurality of operational phases of the piece ofequipment.

E3. The data processing system of any of E0-E2, wherein each of theplurality of aircraft has a shared plurality of operational phases, andthe plurality of instructions are further executable by the one or moreprocessors to:

divide the operational data into a plurality of subsets corresponding tothe shared plurality of operational phases.

E4. The data processing system of E3, wherein the shared plurality ofoperational phases are phases of flight.

E5. The data processing system of E4, wherein the shared plurality ofoperational phases include taxi-in, takeoff, climb, cruise, descent,landing, and taxi-out.

E6. The data processing system of E4 or E5, wherein the operational datainclude a series of sensor readings from an aircraft of plurality ofaircraft, the series of sensor readings corresponding to a flight, andthe plurality of instructions are further executable by the one or moreprocessors to:

display the series of sensor readings separate from the timeline, inresponse to an interaction with the graphical user interface.

E7. The data processing system of any of E3-E6, wherein the plurality ofinstructions are further executable by the one or more processors to:

receive a selection of operational phases, and

display the operational data subsets corresponding to the selectedoperational phases on the timeline in the graphical user interface.

E8. The data processing system of E7, wherein the displayed operationaldata subsets are color-coded according to the corresponding operationalphase.

E9. The data processing system of any of E0-A8, wherein displaying theoperational data includes color-coding the data according to time beforeor after a maintenance event

E10. The data processing system of any of E0-E9, wherein displaying themaintenance data includes displaying additional information related to amaintenance event, in response to an interaction of the user with thedisplayed maintenance event.

E11. The data processing system of any of E0-E10, wherein the selectedaircraft include a plurality of aircraft.

Advantages, Features, and Benefits

The different examples of the predictive maintenance model design systemdescribed herein provide several advantages over known solutions forusing machine learning models to forecast maintenance. For example,illustrative examples described herein allow creation of models foreffective maintenance forecasting without programming or data scienceexpertise.

Additionally, and among other benefits, illustrative examples describedherein facilitate identification of relevant data patterns and precursorsignatures in a large and complex dataset.

Additionally, and among other benefits, illustrative examples describedherein allow long term trending of phase-dependent operational data.

Additionally, and among other benefits, illustrative examples describedherein allow identification and elimination of seasonal bias, which inturn improves effectiveness of generated predictive maintenance modelsby reducing false positive alerts.

Additionally, and among other benefits, illustrative examples describedherein facilitate rapid visualization of a wide variety of data featuresfor a variety of data subsets and in a variety of plot types.

No known system or device can perform these functions, particularly in acomplex dataset based on recorded telemetry from equipment cyclingthrough multiple operational phases. Thus, the illustrative examplesdescribed herein are particularly useful for aircraft maintenanceforecasting. However, not all examples described herein provide the sameadvantages or the same degree of advantage.

Conclusion

The disclosure set forth above may encompass multiple distinct exampleswith independent utility. Although each of these has been disclosed inits preferred form(s), the specific examples thereof as disclosed andillustrated herein are not to be considered in a limiting sense, becausenumerous variations are possible. To the extent that section headingsare used within this disclosure, such headings are for organizationalpurposes only. The subject matter of the disclosure includes all noveland nonobvious combinations and subcombinations of the various elements,features, functions, and/or properties disclosed herein. The followingclaims particularly point out certain combinations and subcombinationsregarded as novel and nonobvious. Other combinations and subcombinationsof features, functions, elements, and/or properties may be claimed inapplications claiming priority from this or a related application. Suchclaims, whether broader, narrower, equal, or different in scope to theoriginal claims, also are regarded as included within the subject matterof the present disclosure.

What is claimed is:
 1. A data processing system for generatingpredictive maintenance models, comprising: one or more processors; amemory including one or more digital storage devices; and a plurality ofinstructions stored in the memory and executable by the one or moreprocessors to: receive a historical dataset relating to each system of aplurality of systems, the historical dataset including time-labeledmaintenance and operational data, receive a first selection of a firstoperational data feature and a first system, display operational dataassociated with the first operational data feature and the first systemon a timeline in a graphical user interface, display maintenance dataassociated with the first system on the timeline, receive a secondselection of a second operational data feature, and generate apredictive maintenance model, using the second operational data featureaccording to a machine learning method.
 2. The data processing system ofclaim 1, wherein the timeline is a time axis of a scatterplot, and theoperational data associated with the first operational data feature andthe first system are displayed as points of the scatterplot.
 3. The dataprocessing system of claim 2, wherein the plurality of systems are afleet of aircraft, and each point of the scatterplot indicates a valuecalculated from a series of sensor readings recorded on a flight.
 4. Thedata processing system of claim 1, wherein the plurality of systems havea shared plurality of operational phases, and the plurality ofinstructions are further executable by the one or more processors to:divide the operational data into a plurality of phase-dependent datasubsets corresponding to the phases of the shared plurality ofoperational phases.
 5. The data processing system of claim 4, whereinthe first selection includes an operational phase, and the displayedoperational data is in a phase-dependent data subset corresponding tothe operational phase.
 6. The data processing system of claim 4, whereinthe plurality of systems are a fleet of aircraft, and the sharedplurality of operational phases are phases of flight.
 7. The dataprocessing system of claim 6, wherein the shared plurality ofoperational phases include one or more of taxi-out, takeoff, climb,cruise, descent, landing, and taxi-in.
 8. The data processing system ofclaim 6, wherein the operational data of the historical dataset includesa plurality of series of sensor readings, each series of sensor readingshaving been recorded during a flight of an aircraft of the fleet ofaircraft, and the plurality of instructions are further executable bythe one or more processors to: receive a selection of a flight, anddisplay the series of sensor readings recorded during the selectedflight, on a separate timeline.
 9. The data processing system of claim1, wherein the first operational data feature is an aggregate datafeature, calculated according to an aggregating statistical function.10. The data processing system of claim 9, wherein displaying theoperational data associated with the first operational data featureincludes displaying a calculated value of the aggregating statisticalfunction for each of a plurality of series of sensor readings.
 11. Thedata processing system of claim 9, wherein receiving the first selectionof the first data feature includes receiving a selection of a sensordata parameter, a position, and the aggregating statistical function,and the aggregate data feature is calculated from operational dataassociated with the sensor data parameter and the position according tothe selected aggregating statistical function.
 12. The data processingsystem of claim 1, wherein the plurality of instructions are furtherexecutable by the one or more processors to: calculate a best-fitperiodic curve for the operational data associated with the firstoperational data feature and the first system, compare a period of thecalculated curve to a year, and when the period is within apre-determined threshold from a year, display the calculated curve onthe timeline of the graphical user interface.
 13. The data processingsystem of claim 12, wherein the plurality of instructions are furtherexecutable by the one or more processors to: modify the operational dataassociated with the first operational data feature and the first systemto compensate for the calculated curve, prior to displaying theoperational data on the timeline of the graphical user interface. 14.The data processing system of claim 1, wherein the first selectionincludes a plurality of systems, and the plurality of instructions arefurther executable by the one or more processors to: display operationaldata associated with each system of the plurality of systems on thetimeline, the operational data associated with each system of theplurality systems being visually distinct from operational dataassociated with the other systems of the plurality of systems.
 15. Thedata processing system of claim 1, wherein displaying maintenance dataassociated with the first system includes displaying a maintenance eventfor the first system, and the plurality of instructions are furtherexecutable by the one or more processors to: visually indicate adivision of the displayed operational data into multiple time periodsaccording to a selected time relationship to the displayed maintenanceevent.
 16. The data processing system of claim 15, wherein the multipletime periods include a first period of time more than a first thresholdtime prior to the displayed maintenance event, and a second period oftime less than a second threshold time prior to the displayedmaintenance event, and the plurality of instructions are furtherexecutable by the one or more processors to: receive a selection of thefirst and second threshold times.
 17. The data processing system ofclaim 1, wherein the plurality of instructions are further executable bythe one or more processors to: receive a selection of two groups ofseries of sensor readings, and display values of the first data featurein one or more of a density plot, a box plot, and a scatter plotincluding a linear regression.
 18. The data processing system of claim17, wherein the first selection includes a plurality of operational datafeatures, and the plurality of instructions are further executable bythe one or more processors to: visually indicate a division of thedisplayed operational data into multiple time periods according to aselected time relationship to a displayed maintenance event, themultiple time periods including a first period of time more than a firstthreshold time prior to the displayed maintenance event, and a secondperiod of time less than a second threshold time prior to the displayedmaintenance event, and wherein a first group of the two groups of seriesof sensor readings is selected from the first time period, and a secondgroup of the two groups of series of sensor readings is selected fromthe second time period.
 19. A computer implemented method of generatinga predictive maintenance model, comprising: receiving a historicaldataset relating to each system of a plurality of systems, thehistorical dataset including time-labeled maintenance and operationaldata, receiving a first selection of a first operational data featureand a first system, displaying operational data associated with thefirst operational data feature and the first system on a timeline in agraphical user interface, displaying maintenance data associated withthe first system on the timeline, receiving a second selection of asecond operational data feature, and generating a predictive maintenancemodel, using the second operational data feature according to a machinelearning method.
 20. A computer program product for generatingpredictive maintenance models, the computer program product comprising:a non-transitory computer-readable storage medium havingcomputer-readable program code embodied in the storage medium, thecomputer-readable program code configured to cause a data processingsystem to generate a predictive maintenance model, the computer-readableprogram code comprising: at least one instruction to receive ahistorical dataset relating to each system of a plurality of systems,the historical dataset including time-labeled maintenance andoperational data, at least one instruction to receive a first selectionof a first operational data feature and a first system, at least oneinstruction to display operational data associated with the firstoperational data feature and the first system on a timeline in agraphical user interface, at least one instruction to displaymaintenance data associated with the first system on the timeline, atleast one instruction to receive a second selection of a secondoperational data feature, and at least one instruction to generate apredictive maintenance model, using the second operational data featureaccording to a machine learning method.